Repository navigation
feat(node): per-IP write-surface rate brake + purge-spam admin tool - #196
Conversation
…utes Introduces a dedicated write_rate_limiter bucket (GITLAWB_WRITE_RATE_LIMIT, default 600/hr, 0 disables), separate from the creation brake so a write flood cannot drain the creation budget (KTD-1). Attaches rate_limit_by_ip to write_routes, mirroring creation_routes. Per-DID deliberately not paired (a DID farm never trips it). Tests (route-level, execution-verified against Postgres): a write_routes sink (PUT /star) is 429'd before auth; an exhausted write bucket does not throttle repo creation (separate-bucket property).
Attaches rate_limit_by_ip (write bucket) to task_write_routes, issue_write_routes, bounty_write_routes, profile_write_routes, and the /graphql POST/GET surface (KTD-5 — an HTTP-layer brake counts every /graphql request; /graphql/ws subscriptions stay unbraked). write_ip_limiter is now built once and shared. Every non-git, non-creation authenticated write group is braked. Tests (execution-verified against Postgres): /graphql POST is 429'd, and a representative REST write sink (issue comment) is 429'd, by the write brake.
…rake Middleware-level tests complementing the client_key unit coverage: a spoofer varying the client-controlled leftmost X-Forwarded-For hop cannot rotate its bucket key (keyed on the trusted rightmost hop); distinct trusted client hops get distinct buckets (no cross-starvation); and a request with no trusted header in a proxy mode falls back to the socket-peer key — the documented collapse, asserted by name so any change is deliberate. Operator-warning on missing-header deferred deliberately: a per-request warning is a log-noise risk and the security-load-bearing fallback (peer key, never skip) is what these tests lock down.
Adds a purge-spam subcommand (clap #[command(subcommand)] split; no-subcommand path still runs the node unchanged). Candidate selection is signature-based, not per-DID: a repo qualifies only if owned by the named burst DID AND verified empty (zero git refs) per repo. A hard exclusion gate runs BEFORE the empty check, so an empty repo owned by the content-bearing user or the mirror-bot DID is never a candidate. Dry-run is the default; deletion needs an explicit --execute flag and operates per-repo, never 'all repos of DID X'. Tests (RED->GREEN, exclusion gate confirmed load-bearing by targeting an excluded DID directly): empty target listed; target-with-refs spared; empty excluded/ intern repos spared (must-assert); dry-run deletes nothing; execute removes only the empty target on disk.
P1 from code review: on_disk_ref_count shelled 'git for-each-ref' with only the repo path as cwd and no --git-dir, so a path that exists but is not a bare repo let git discover a parent .git (repos_dir may sit inside the operator's checkout) and read a DIFFERENT repo's refs — possibly 0, deleting a real repo. Now require the bare-repo markers (HEAD file + objects/ dir) before trusting any count; anything else fails closed (skipped). Tests: a non-git dir under a git ancestor (zero-ref) now fails closed — verified RED without the guard (read the ancestor's 0 refs); target-DID-never-excluded invariant pinned; write_flood test anchored to a genuinely-drained bucket so its assert_ne can't pass vacuously.
…rage Addresses the remaining code-review findings: - purge-spam DID matching now uses did:key normalization on both sides (list_repos_by_owner_did via OWNER_KEY_CASE_SQL, is_excluded and the target scope via normalize_owner_key), so short/full form is consistent and an excluded identity is protected in either form. Adds a short-form exclusion test. - Extract the per-IP write brake into a chainable WriteBraked helper so a future write group can't silently ship unbraked (kept separate from add_auth_layers by design; documented why). - Mark --repos-dir / --database-url global so admin subcommands accept them after the subcommand (was an 'unexpected argument' parse error); verified. - Document GITLAWB_WRITE_RATE_LIMIT in .env.example incl. the per-IP NAT-aggregate caveat so operators can size it. - Coverage: adoption-floor (under-limit write passes), write-limit=0 disables, task/bounty/profile route-level 429s, /graphql/ws stays unbraked. Full suite 927 pass / 0 fail; fmt + clippy -D warnings clean.
- purge-spam execute now re-verifies emptiness immediately before delete (TOCTOU: a push between selection and delete no longer risks deleting a now-non-empty repo) and removes the on-disk bare repo dir so DB and disk stay consistent; a per-repo delete failure warns and continues rather than aborting the batch. - Tests (RED-verified where they cover new behavior): partition_for_delete skips a repo no longer empty; execute removes the on-disk dir (RED without the removal); list_repos_by_owner_did finds a short-form burst row (RED without the SQL normalization); every braked write group passes under-limit (per-group grant path, not just the star representative).
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
Walkthrough
Merge Risk:
|
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
crates/gitlawb-node/src/main.rs (1)
440-455: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
write_rate_limiteris missing from the periodic cleanup loop.Every other limiter (
rate_limiter,create_ip_rate_limiter,push_rate_limiter,sync_trigger_rate_limiter,peer_write_rate_limiter) is swept here, but the newly addedwrite_rate_limiteris not. It is bounded, so no unbounded leak, yet stale keys are only evicted lazily on a capacity miss instead of proactively like its peers — an inconsistency worth closing.🧹 Proposed fix
let peer_write_rl = state.peer_write_rate_limiter.clone(); + let write_rl = state.write_rate_limiter.clone(); let db = state.db.clone();peer_write_rl.cleanup().await; + write_rl.cleanup().await;🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/gitlawb-node/src/main.rs` around lines 440 - 455, Include the state.write_rate_limiter in the periodic cleanup task alongside the other cloned rate limiters. Clone it before spawning the task and invoke its cleanup method in the five-minute tokio::select! interval, preserving the existing cleanup behavior for all other limiters.
🧹 Nitpick comments (1)
crates/gitlawb-node/src/api/repos.rs (1)
2629-2966: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider extracting a shared test-request helper to reduce duplication.
All 9 new tests repeat the same
Request::builder()+ConnectInfoinsertion +oneshot()+ status-assert boilerplate.rate_limit.rsin this same PR already factored this pattern intopost_from/post_withhelpers — a similar helper here (e.g.,send_from(&router, method, uri, body, peer)) would cut most of the repetition acrosswrite_route_is_rate_limited_by_ip,write_flood_does_not_drain_creation_budget,graphql_post_is_rate_limited_by_ip,issue_comment_is_rate_limited_by_ip,under_limit_write_is_not_throttled,write_rate_limit_zero_disables_the_brake,task_bounty_profile_writes_are_rate_limited,graphql_ws_is_not_braked, andevery_write_group_passes_under_limit.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/gitlawb-node/src/api/repos.rs` around lines 2629 - 2966, Reduce duplicated request setup across the listed rate-limit tests by adding a shared test helper such as send_from that accepts the router, HTTP method, URI, body, and peer, inserts ConnectInfo, sends via oneshot, and returns the response or status. Update each named test to use this helper while preserving its existing request bodies, methods, URIs, and status assertions.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@crates/gitlawb-node/src/main.rs`:
- Around line 440-455: Include the state.write_rate_limiter in the periodic
cleanup task alongside the other cloned rate limiters. Clone it before spawning
the task and invoke its cleanup method in the five-minute tokio::select!
interval, preserving the existing cleanup behavior for all other limiters.
---
Nitpick comments:
In `@crates/gitlawb-node/src/api/repos.rs`:
- Around line 2629-2966: Reduce duplicated request setup across the listed
rate-limit tests by adding a shared test helper such as send_from that accepts
the router, HTTP method, URI, body, and peer, inserts ConnectInfo, sends via
oneshot, and returns the response or status. Update each named test to use this
helper while preserving its existing request bodies, methods, URIs, and status
assertions.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: e186f361-729a-419c-b633-a90e12b77a44
📒 Files selected for processing (11)
.env.examplecrates/gitlawb-node/src/admin.rscrates/gitlawb-node/src/api/repos.rscrates/gitlawb-node/src/auth/mod.rscrates/gitlawb-node/src/config.rscrates/gitlawb-node/src/db/mod.rscrates/gitlawb-node/src/main.rscrates/gitlawb-node/src/rate_limit.rscrates/gitlawb-node/src/server.rscrates/gitlawb-node/src/state.rscrates/gitlawb-node/src/test_support.rs
jatmn
left a comment
There was a problem hiding this comment.
High — fix before merge
H1. Read/fork handlers run blocking git subprocesses directly on the tokio runtime (no spawn_blocking, no timeout)
fork_repo—crates/gitlawb-node/src/api/repos.rs:1567-1575invokesstd::process::Command::new("git").args(["clone","--mirror",…]).output()inside the async handler.output()blocks the calling worker thread;git clone --mirrorof a non-trivial repo can take minutes and there is no timeout, unlike every other git walk in this file which correctly usestokio::task::spawn_blocking(repos.rs:67, 110, 661, 1143) ortokio::process::Command(info/refs, upload-pack, receive-pack).- Read metadata handlers —
list_commits/get_tree/get_blobcallstore::{resolve_head, log, ls_tree, read_file}(api/repos.rs:360-361, 397, 443, 481) which are synchronousstd::process::Command::output()calls (crates/gitlawb-node/src/git/store.rs:92-231). A slow disk or large repo blocks a worker thread for seconds; with a finite worker pool a handful of concurrent/commits,/tree,/blob, or a single/forkcan starve the whole runtime, including/healthand/ready(same runtime inbuild_router). - Impact: availability/reliability regression, not just latency.
git_service_timeout_secsis bypassed on these paths. - Fix: wrap the blocking
storecalls (and the fork clone) intokio::task::spawn_blockingand bound withtokio::time::timeout(git_service_timeout_secs, …), returningAppError::Timeout.
H2. write_rate_limiter is never reaped by the background cleanup loop
crates/gitlawb-node/src/main.rs:440-455— the periodic task clones exactly five limiters (rate_limiter,create_ip_rate_limiter,push_rate_limiter,sync_trigger_rate_limiter,peer_write_rate_limiter) and calls.cleanup()on each.state.write_rate_limiter(declared atmain.rs:329-333, wired intoAppStateat:404) is omitted, even though it protects the entire authenticated non-creation write surface +/graphql(server.rs:84-98, 110, 154+).- The
RateLimiterdesign makes the inline capacity sweep amortized and explicitly relies on the background loop for full reclamation (rate_limit.rs:36-39, 94-114). With no background sweep,write_rate_limiteronly reclaims expired keys once its map hits the 200,000-key cap. Until then every distinct client IP persists as a staleWindowfor the full 1-hour window and beyond → steady memory growth (bounded at the cap) and a latent correctness cliff: once at cap, the amortized inline sweep becomes the only reclaimer and a distinct-IP write flood serializes behind O(max_keys) scans while legitimate new IPs get spuriously 429'd between sweeps. - Fix: add
let write_rl = state.write_rate_limiter.clone();andwrite_rl.cleanup().await;inside the cleanup tick beside the other five.
Medium — should fix
M1. fork_repo / create_repo orphan on-disk repos (and a spent proof / uploaded object) on late DB failure
fork_repoordering (api/repos.rs:1555 → 1567 → 1585 → 1605): spend proof →git clone --mirrortodisk_path→release_after_write(Tigris upload) →db.create_repo. Ifcreate_repofails, the on-disk mirror and the already-uploaded Tigris object are left with no DB row, and the single-use iCaptcha proof is already consumed.create_repo(api/repos.rs:208 → 210-219 → 236):proof.consumethenrepo_store.init(blockinggit init) happen beforedb.create_repo. A DB failure orphans the freshly-initialized directory.inititself is also a blocking git call on the async runtime (same class as H1).- Fix: make disk materialization and proof spend adjacent to a guaranteed-successful DB write, and on
db.create_repofailure best-effortremove_dir_all(disk_path)(and prefer deferring the Tigris upload until after the row commits).
M2. TrustedProxy IP trust is config-asserted, not network-enforced — header rotation defeats every per-IP brake
crates/gitlawb-node/src/rate_limit.rs:198-218(client_key): inFly/XForwardedFormodes the limiter key is taken entirely fromFly-Client-IPor the rightmostX-Forwarded-Forhop, and only falls back to the socket peer when the header is absent. There is no check that the socket peer (ConnectInfo) is actually the operator's proxy. IfGITLAWB_TRUSTED_PROXYis set but the node's HTTP port is reachable directly (misconfigured firewall, exposed Fly port, second ingress, internal attacker), a client can setFly-Client-IP: <random>/ append any rightmostX-Forwarded-Forhop and get a fresh bucket per request, evading the creation/write/push/sync brakes (the only thing stopping a single-source spam/push flood). The unit tests cover header parsing but never assert "peer not the proxy → key on peer."- Fix: honor the forwarded header only when
ConnectInfopeer is in a configured proxy CIDR/IP allowlist; otherwise key on the socket peer (fail closed). Add a test for "trust mode set, request from a non-proxy IP." - Note (validation): exploitability is deployment-dependent (Fly's edge does overwrite
Fly-Client-IP), so rated Medium not Blocker.
M3. purge-spam --execute deletes via remove_dir_all on an unsanitized repo_name (path traversal → arbitrary repo deletion)
crates/gitlawb-node/src/git/store.rs:450-453—repo_disk_pathsanitizesowner_did(replace([':', '/'], "_")) but joinsrepo_nameverbatim viaformat!("{repo_name}.git")with no../slash/dot validation.crates/gitlawb-node/src/admin.rs:243-244callsstd::fs::remove_dir_all(store::repo_disk_path(&repos_dir, &c.owner_did, &c.name))on that path. The fail-closed marker guard (admin.rs:128) only requires aHEADfile +objects/dir — i.e. any existing legitimate bare repo satisfies it. API names are validated (api/repos.rs:189-197), butsmart_http_repo_nameexplicitly notes repos are "creatable via the peer mirror path, which skips API name validation" (api/repos.rs:490-494), so a burst-DID row with a../name can be planted, then deleted outsiderepos_dir.- Fix: validate
repo_name(and re-validateowner_did) the same way the serving paths do (repo_store::validate_repo_name/validate_path_components), and assert the resolved path is canonically insiderepos_dirbefore anyremove_dir_all. Ideally makerepo_disk_pathitself reject unsafe names. - Note: a destructive
remove_dir_allshould never run on a DB-sourced, unsanitized name. End-to-end exploitability is narrow (requires the burst DID + marker check at the traversed path), so rated Low/Medium — but it is a clear defense-in-depth defect.
M4. purge-spam TOCTOU: emptiness re-checked once as a batch snapshot, not immediately before each delete — races a live push
crates/gitlawb-node/src/admin.rs:221-235—partition_for_delete(:224-226) re-verifies emptiness for all candidates up front, producingto_delete; the actualdelete_repo_by_idruns later in a loop (:235). Between the snapshot and a given delete, a concurrentgit-receive-packon the running node can add a ref, making the repo non-empty — yet it is still deleted (silent data loss). The comment claims it "re-verify[s] emptiness immediately before deleting," but the recheck is a single batch pass, not a per-candidate check immediately beforedelete_repo_by_id.- The execute path also does not acquire the node's
acquire_writeadvisory lock (api/repos.rs:927), so it races a live push: a push can leave orphaned Tigris objects / ref-certificate rows referencing a now-deletedrepos.id, andremove_dir_allcan race an in-flightreceive-packwritingobjects/. - Fix: re-verify
ref_count_on_diskimmediately before eachdelete_repo_by_id(or wrap recheck+delete of each repo in arepo_store.acquire_writeguard / DB transaction re-checking a ref signal). Document that the tool must be run against a quiesced node for the target DID.
M5. iCaptcha gate fails open in enforce mode when the public key cannot load
crates/gitlawb-node/src/icaptcha.rs:302-304— insidedecide,if v.key.is_none() { return Decision::Allow; }runs before the mode match, so inMode::Enforcea missing key (iCaptcha unreachable at startup, DNS/BGP block, badICAPTCHA_PUBKEY) allows every create/registration request with no proof.init()(icaptcha.rs:160-166) leaveskey = Noneand only warns. Since the gate defaults tooff, the default deployment already has no captcha; enforce-mode is now trivially bypassable by denying the key fetch, directly weakening the abuse protection this PR leans on after PoW removal.- Fix: in enforce mode fail closed (reject) when
key.is_none(); or refuse to start in enforce mode without a key. At minimum document that enforce+no-key == fail-open.
M6. purge-spam deletes the DB row, then removes the dir; a disk failure is swallowed while the deleted counter is already incremented
crates/gitlawb-node/src/admin.rs:235-248— onOk(n)the row is counted asdeletedbeforeremove_dir_all, which on error onlywarn!s. The final "deleted N repo row(s)" summary therefore reports success even when the on-disk repo survives → DB/disk drift and a misleading operator-facing count. No retry / non-zero exit signal.- Fix: remove the on-disk dir first (or track disk-failure separately from the DB-deleted count) and surface "N deleted, M failed."
M7. delete_repo_by_id (and purge-spam) leave dependent rows orphaned
crates/gitlawb-node/src/db/mod.rs:1288-1294—DELETE FROM repos WHERE id = $1with no cascade. Child tables keyed onrepo_id(branch_cids,ref_certificates,visibility_rules,repo_stars,encrypted_blobs,received_ref_updates,repo_replicas,arweave_anchors,webhooks,pull_requests) keep orphan rows that accumulate and can be re-joined if a repo with the same id is later created.- Fix: add
ON DELETE CASCADE(or explicit child deletes) for therepo_idFKs, ideally within the same transaction as the row delete.
Low / Info — nice to have
- L1. Rate-limiter bounded-map DoS.
rate_limit.rs:96-114rejects new keys when at the 200k cap; a distinct-key flood (IPv6 variation or the M2 header rotation) then 429s legitimate new clients for up to a full window. Consider evicting expired entries to admit new keys, or a metric on cap-full rejections. - L2. Limiter state is per-process / in-memory (
rate_limit.rs:24-41). In multi-machine Fly deployments the effective per-IP/per-DID limit isN × configuredand a restart wipes buckets. Known tradeoff — document explicitly; consider shared counters for the spam-relevant paths. - L3. Non-atomic
create_repo.api/repos.rs:203-236check-then-insert with noUNIQUE(owner_did, name); theidx_repos_owner_key_nameis a plain index (db/mod.rs:474-490). A concurrent create race surfaces a rawUNIQUE(disk_path)violation mapped to a generic 500 instead ofRepoExists(409). - L4. Read handlers mask git errors as empty
200 OK.api/repos.rs:361(store::log(...).unwrap_or_default())and theget_treesiblings coerce real backend failures into empty lists, unlikeget_blob(api/repos.rs:398-410) which classifies errors. MapErrtoAppError::Git/NotFound. - L5.
parse_ref_updatessilently tolerates malformed/truncated bodies (api/repos.rs:1651-1699) → emptyVec<RefUpdate>; receive-pack then proceeds with zero ref updates (skipping branch-protection/owner-push checks). Return400on a non-empty body that parses short. - L6. Push bookkeeping errors swallowed (
api/repos.rs:959, 977, 982, 999-1005let _ =/ only-warn!) — transient DB blips silently loserecord_push/update_trust_score. At least raise log severity / add a durable outbox. - L7.
git_info_refsreceive-pack brake skipped whenclient_keyreturnsNone(api/repos.rs:556/rate_limit.rs:198): if no trusted header and noConnectInfopeer, theif let Some(key)guard is skipped and the request proceeds unthrottled. Unreachable in prod (socket peer always present) but a divergent contract fromrate_limit_by_ip; document or reject instead of skip. - L8. Malformed rate-limit env vars silently fall back to defaults.
main.rs:304-307, 325-328, 343-346parseGITLAWB_CREATE/WRITE/PUSH_RATE_LIMITwith.ok().and_then(parse).unwrap_or(default); a fat-fingered value silently disables the intended (stricter) brake with no warning, unlike the clap-validated siblings. Warn (or fail fast) on present-but-unparseable values. - L9.
.env.exampledocs drift. MissingGITLAWB_IPFS_API,GITLAWB_TIGRIS_BUCKET,GITLAWB_METRICS_ADDR,GITLAWB_SHUTDOWN_GRACE_SECS(parsed inconfig.rs:88,161,188,194). Reverse direction is clean. Add with defaults. - L10. Weak dry-run safety test.
admin.rs:471-492(dry_run_deletes_nothing) seeds repos with an emptyrepos_dir, sorun_purge_spamhits the no-candidate early return before any delete logic — it does not exercise theif !execute { return }guard when candidates exist. Add a dry-run test with a real empty on-disk target repo so a regression that deletes in dry-run is caught. - L11. Unauthenticated
/metrics(main.rs:828-875,config.rs:187warns it is unauthenticated). If bound to a non-loopback addr it leaks peer/repo/push counts. Operator-controlled; note in hardening guidance. - L12. Git error strings returned to clients (
AppError::Git(e.to_string())sites, e.g.api/repos.rs:398-410). Can surface internal paths; return opaque messages, keep detail in server-side logs.
…ve_dir_all A peer-mirror row skips API name validation, so a burst-owned repo can carry a '../' name; repo_disk_path joined it verbatim and the emptiness marker check passed on the traversed target, so purge-spam selected and remove_dir_all'd a repo OUTSIDE repos_dir. Validate the name via repo_store::validate_repo_name at selection (fail closed -> non-candidate) and assert canonical containment inside repos_dir immediately before the destructive remove. Adversarial RED->GREEN: a victim bare repo reachable via '../../victim' is deleted before, survives after; symlink-escape covered by path_within.
delete_repo_by_id was a bare DELETE FROM repos with no cascade and the schema has no FKs, so every child row (ref_certificates, pull_requests + pr_reviews/ pr_comments, webhooks, push_events, protected_branches, repo_stars, repo_replicas, repo_labels, visibility_rules, encrypted_blobs, repo_icaptcha_proofs, agent_tasks, and the slug-keyed branch_cids/sync_queue/received_ref_updates/arweave_anchors) was orphaned and could re-join a later repo reusing the id. Wrap parent+child deletes in one transaction, resolving the owner_short/name slug for the repo-TEXT children. bounties (financial) and issue_comments (unmappable) are left intact by design. RED->GREEN test seeds a repo_id child, a slug child, and a PR grandchild.
The PR's new write_rate_limiter was the one limiter omitted from main()'s hand-maintained cleanup list, so its key map never shed expired entries until the 200k cap. Move the sweep into AppState::cleanup_rate_limiters (co-located with the limiter fields, single source of truth) and call it from the loop, so a newly-added limiter is reaped rather than silently dropped. Guard test seeds a reclaimable entry into write_rate_limiter and asserts the sweep reaps it — RED if write_rate_limiter is dropped from cleanup_rate_limiters (verified).
purge-spam took no write lock, so a concurrent receive-pack could land a ref between the emptiness recheck and the delete, silently losing a live push. Add RepoStore::try_lock_repo — a lock-only, non-blocking counterpart to acquire_write (no Tigris I/O) whose guard holds a dedicated pooled connection so the advisory lock and its unlock run on the same session. purge now locks each candidate, rechecks emptiness UNDER the lock, deletes, and releases; a repo held by a live writer is skipped, not force-deleted. RED->GREEN: a locked empty repo survives purge, then deletes once the writer releases (neuter-check confirms the lock is load-bearing).
…etes The Ok(n) arm incremented the deleted total before remove_dir_all, whose failure only warn!'d, so the summary reported N deleted even when the on-disk dir survived (DB/disk drift shown as clean success). run_purge_spam now returns a PurgeSummary and counts a failed (or containment-refused) on-disk removal in disk_failed, never folding it into deleted; the summary line surfaces 'M on-disk removal(s) failed'. RED->GREEN: a read-only parent dir forces the removal to fail; deleted=1 AND disk_failed=1 with the dir surviving (neuter-check confirms the counter is load-bearing).
…y defaulting GITLAWB_CREATE/WRITE/PUSH_RATE_LIMIT parsed with .ok().and_then(parse).unwrap_or, so a typo (e.g. 5O with a letter O) silently reverted to the default and left the operator believing a stricter cap was in force. Route all three through rate_limit_from_env, which distinguishes absent/empty (default, silent) from present-but-unparseable (default, WARN). Pure resolve_rate_limit is unit-tested across absent/empty/valid/zero/unparseable — 0 stays valid (its own ==0 warn handles the disable case).
…DOWN_GRACE_SECS in .env.example Four config knobs read in config.rs were absent from .env.example, so operators had no template for them. Add each in its matching section with the config default (INV-24: operator docs match the config the code reads, in the same PR that touches the file).
…present
dry_run_deletes_nothing ran against an empty repos_dir, so select_spam_candidates
found nothing and run_purge_spam hit the no-candidate early return — the
if !execute { return } guard was never exercised with candidates present (a
vacuous guard, INV-21). Materialize a real empty bare repo so a genuine candidate
exists, then assert dry-run leaves both the row and the on-disk dir. Verified
load-bearing: removing the !execute guard turns the test RED (dry-run deletes).
|
Pushed the fixes at Fixed in this PR
Deferred to follow-up issues (pre-existing, outside this PR's diff)
Declined
@jatmn ready for another look. |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.env.example (1)
143-152: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winDocument all GraphQL HTTP requests covered by this limiter.
The route wiring in
crates/gitlawb-node/src/server.rsapplies the brake to the whole/graphqlroute, so queries and the playground GET also consume this bucket—not only mutations. Update the text to say “GraphQL HTTP requests” or narrow the middleware.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.env.example around lines 143 - 152, Update the write rate-limiting documentation near GITLAWB_WRITE_RATE_LIMIT to state that the limiter covers all GraphQL HTTP requests on the /graphql route, including queries and the playground GET, rather than describing only GraphQL mutations. Keep the existing rate-limit scope and configuration details unchanged.
🧹 Nitpick comments (3)
.env.example (1)
19-19: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winKeep the dotenv keys in linter order.
dotenv-linterreports thatGITLAWB_METRICS_ADDRshould appear beforeGITLAWB_PORT; moving it will keep configuration checks clean.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.env.example at line 19, Move the GITLAWB_METRICS_ADDR entry in .env.example so it appears before GITLAWB_PORT, preserving the existing key definitions and values.Source: Linters/SAST tools
crates/gitlawb-node/src/main.rs (1)
292-345: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExtract the repeated per-IP limiter setup (create/write/push) into a helper.
The create/write/push blocks each repeat "resolve env limit → build
RateLimiter::new_bounded→ warn if zero" verbatim, just with different var names/messages. A small helper collapses the duplication and makes adding a fourth limiter trivial.♻️ Proposed refactor
+fn build_ip_limiter(limit: usize, var: &str, label: &str) -> rate_limit::RateLimiter { + let limiter = rate_limit::RateLimiter::new_bounded( + limit, + std::time::Duration::from_secs(3600), + 200_000, + ); + if limit == 0 { + tracing::warn!("{var}=0 — per-IP {label} rate limiting disabled"); + } + limiter +} + let create_limit = rate_limit_from_env("GITLAWB_CREATE_RATE_LIMIT", 120); - let create_ip_rate_limiter = rate_limit::RateLimiter::new_bounded( - create_limit, - std::time::Duration::from_secs(3600), - 200_000, - ); - if create_limit == 0 { - tracing::warn!("GITLAWB_CREATE_RATE_LIMIT=0 — per-IP creation rate limiting disabled"); - } + let create_ip_rate_limiter = + build_ip_limiter(create_limit, "GITLAWB_CREATE_RATE_LIMIT", "creation"); let write_limit = rate_limit_from_env("GITLAWB_WRITE_RATE_LIMIT", 600); - let write_rate_limiter = rate_limit::RateLimiter::new_bounded( - write_limit, - std::time::Duration::from_secs(3600), - 200_000, - ); - if write_limit == 0 { - tracing::warn!("GITLAWB_WRITE_RATE_LIMIT=0 — per-IP write rate limiting disabled"); - } + let write_rate_limiter = build_ip_limiter(write_limit, "GITLAWB_WRITE_RATE_LIMIT", "write"); let push_limit = rate_limit_from_env("GITLAWB_PUSH_RATE_LIMIT", 600); - let push_rate_limiter = rate_limit::RateLimiter::new_bounded( - push_limit, - std::time::Duration::from_secs(3600), - 200_000, - ); - if push_limit == 0 { - tracing::warn!("GITLAWB_PUSH_RATE_LIMIT=0 — per-IP push rate limiting disabled"); - } + let push_rate_limiter = build_ip_limiter(push_limit, "GITLAWB_PUSH_RATE_LIMIT", "push");🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/gitlawb-node/src/main.rs` around lines 292 - 345, Extract the repeated create, write, and push per-IP limiter initialization into a shared helper that accepts the environment variable name, default limit, and disabled-warning context, then returns the bounded RateLimiter. Replace the three inline rate_limit_from_env/new_bounded/warn sequences with helper calls while preserving their existing limits, environment overrides, warning behavior, and bucket duration/capacity.crates/gitlawb-node/src/git/repo_store.rs (1)
428-446: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winMake repo lock release drop-safe
RepoLockGuard::release()dropspg_advisory_unlockerrors, and the manual guard still depends on every caller remembering.release(). Switching tosqlx::postgres::PgAdvisoryLock/PgAdvisoryLockGuard(or adding a logged, drop-safe fallback) would make leaked locks much harder to miss.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/gitlawb-node/src/git/repo_store.rs` around lines 428 - 446, Update RepoLockGuard and RepoLockGuard::release so advisory locks are drop-safe: use sqlx’s PgAdvisoryLock/PgAdvisoryLockGuard where compatible, or add a Drop-based fallback that attempts cleanup and logs unlock failures. Ensure callers no longer depend solely on remembering release(), while preserving unlocking on the same connection.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In @.env.example:
- Around line 143-152: Update the write rate-limiting documentation near
GITLAWB_WRITE_RATE_LIMIT to state that the limiter covers all GraphQL HTTP
requests on the /graphql route, including queries and the playground GET, rather
than describing only GraphQL mutations. Keep the existing rate-limit scope and
configuration details unchanged.
---
Nitpick comments:
In @.env.example:
- Line 19: Move the GITLAWB_METRICS_ADDR entry in .env.example so it appears
before GITLAWB_PORT, preserving the existing key definitions and values.
In `@crates/gitlawb-node/src/git/repo_store.rs`:
- Around line 428-446: Update RepoLockGuard and RepoLockGuard::release so
advisory locks are drop-safe: use sqlx’s PgAdvisoryLock/PgAdvisoryLockGuard
where compatible, or add a Drop-based fallback that attempts cleanup and logs
unlock failures. Ensure callers no longer depend solely on remembering
release(), while preserving unlocking on the same connection.
In `@crates/gitlawb-node/src/main.rs`:
- Around line 292-345: Extract the repeated create, write, and push per-IP
limiter initialization into a shared helper that accepts the environment
variable name, default limit, and disabled-warning context, then returns the
bounded RateLimiter. Replace the three inline
rate_limit_from_env/new_bounded/warn sequences with helper calls while
preserving their existing limits, environment overrides, warning behavior, and
bucket duration/capacity.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 0ed69896-8eea-4a8f-802d-9ceb9db33022
📒 Files selected for processing (7)
.env.examplecrates/gitlawb-node/src/admin.rscrates/gitlawb-node/src/db/mod.rscrates/gitlawb-node/src/git/repo_store.rscrates/gitlawb-node/src/main.rscrates/gitlawb-node/src/rate_limit.rscrates/gitlawb-node/src/state.rs
🚧 Files skipped from review as they are similar to previous changes (1)
- crates/gitlawb-node/src/admin.rs
jatmn
left a comment
There was a problem hiding this comment.
I found issues that need to be addressed before this is ready.
Merge readiness
- All required GitHub checks on head
da876e8491fe42d276a268b95f32eb1bced1a112are passing; branch is mergeable against currentmain(7ae4707c4848a4d8fab75a3edf510e5508d3a8e1). - Prior jatmn findings on release-manifest rollback and
gitlawb-coredependency-purity gate appear resolved on this head (no.release-please-manifest.jsonregression in the three-dot diff; workflow gate retained onmain). - Reconcile merge commit
1459cf2: it dropped durable-upload rollback code that earlier #196 commits on this branch had landed. The P1 item below is not a request to fix novelmain-only debt in isolation — it asks you to re-land your own fix and align comments with behavior.
Findings
🟠 P1 — Re-land close/merge durable-upload rollback (lost in merge; comment now overclaims)
📍 Where: crates/gitlawb-node/src/api/issues.rs (close_issue), crates/gitlawb-node/src/api/pulls.rs (merge_pr), and the git helpers/tests commit 1d8c281 had added (crates/gitlawb-node/src/git/issues.rs; merge tip restore in git/store.rs for merge_pr).
💥 What fails: A local issue close or branch merge succeeds on disk, then RepoWriteGuard::release hits a definite publish refusal (UploadFailed / Fenced). The handler returns 5xx while the local tree keeps the mutation. RepoStore::acquire fast-path can serve that state until a later refresh.
// close_issue on head (comment promises rollback; implementation does not)
guard.release(close_result.is_ok()).await.into_result()?;create_issue in the same file already compensates on refusal:
guard.release_compensating(true, move |path| {
git_issues::delete_issue_ref(&git_bin, path, &issue_id_for_comp, deadline)
})At merge-base 4fc041a, close_issue and merge_pr already used plain release with no compensating closure — the same observable gap. This PR is not inventing a new class of bug; it claimed to close that class on close/merge, implemented it in 1d8c281, then dropped it in merge 1459cf2 while leaving a comment at close_issue that still describes rollback.
🔎 Root cause: This branch’s merge integration removed compensating release, snapshot helpers, and RED tests, without either restoring the behavior or deleting the misleading comment. merge_pr was not re-synced after the same merge.
📜 Stated contract:
create_issueusesrelease_compensatingon publish failure; this branch’s commits5a46ef6/1d8c281and the PR description claim improved close/merge behavior when durable storage fails; the in-diff comment atclose_issuedocuments pre-close OID snapshot and rollback.
🏷️ Attribution: Incomplete claimed fix on this branch (merge regression of your own patch). Underlying close/merge behavior without compensation is unchanged pre-existing at merge-base — not a mandate to scope-creep into unrelated main call sites.
📌 In this PR:
close_issuehandler and the rollback comment block (api/issues.rs)merge_prhandler (api/pulls.rs)- re-apply helpers and tests from
1d8c281(issue_ref_oid,restore_issue_ref,issue_close_failed_upload_rolls_back_to_open, merge analogue)
🔒 Unchanged on main: create_issue compensating pattern, receive-pack path, and Tigris conditional-PUT design — do not redesign upload core.
🔧 Required correction: Cherry-pick/reapply the 1d8c281 shape: capture pre-mutation refs, guard.release_compensating so definite refusal runs rollback while the advisory lock is held, restore helpers and RED tests, and either implement rollback or remove the rollback comment so operator-facing text matches code. Close every listed in-diff row in one pass.
🛠️ Author fix: Re-land the fix this branch already wrote — handlers, helpers, and tests together. Do not patch only the comment or only close_issue.
🚫 Out of scope: Fixing every other write path on main that never used compensation; rebuilding Tigris versioning/conditional-PUT semantics (separate jatmn thread).
🟡 P2 — Purge execute must still delete the object-store archive when disk-path re-validation fails after the DB row is gone
📍 Where: crates/gitlawb-node/src/admin.rs — run_purge_spam execute loop after successful delete_repo_by_id.
💥 What fails: The DB row is deleted and summary.deleted increments, but validated_repo_disk_path then returns Err. The loop continues before delete_archive, leaving the archive in the bucket.
🔎 Root cause: Archive deletion is sequenced after disk-path validation; the error arm exits the iteration without calling repo_store.delete_archive.
📜 Stated contract:
Comments at the execute tail: delete the archive "else it survives and can be downloaded into a later repo created with the same owner/name";
archive_failedexists so survivors are visible.
🏷️ Attribution: PR-introduced (admin.rs is new).
📌 In this PR:
run_purge_spamexecute loop only
🔧 Required correction: On path-validation failure after a successful row delete, still attempt delete_archive (and count archive_failed on error) before continue, or reorder so archive deletion is not skipped when disk removal is refused.
🛠️ Author fix: Close this path in the same change set as any other purge execute fixes — do not fix archive-only while leaving the symlink recheck gap below unaddressed if you touch ref_count_on_disk.
🚫 Out of scope: Redesigning purge to be transactional across DB+disk+S3.
🟡 P2 — Execute-time emptiness recheck must apply the same symlink fail-closed rule as candidate selection
📍 Where: crates/gitlawb-node/src/admin.rs — ref_count_on_disk (used for under-lock recheck and partition_for_delete) vs local_refs_on_disk.
💥 What fails: Selection rejects a symlink at the repo path (symlink_metadata + warn). Between selection and under-lock recheck, the path can become a symlink to an empty bare repo outside repos_dir. ref_count_on_disk follows the link (HEAD/objects via the target), reports 0 refs, and purge deletes the DB row while path_within may refuse disk removal.
🔎 Root cause: Asymmetric guards: symlink rejection exists only on the selection closure, not on the authoritative recheck function.
📜 Stated contract:
Test
symlink_to_outside_empty_bare_fails_closedand module docs: a symlink at the purge target must fail closed and must not be treated as empty-and-purgeable.
🏷️ Attribution: PR-introduced.
📌 In this PR:
ref_count_on_disk- under-lock recheck at execute (
ref_count_on_diskcall) - add execute-path regression test mirroring the selection symlink test
🔧 Required correction: Share one helper or duplicate the symlink_metadata symlink guard from local_refs_on_disk inside ref_count_on_disk so execute cannot admit symlink targets as empty.
🛠️ Author fix: Fix selection and execute siblings together; add the execute symlink test in the same pass.
🚫 Out of scope: Changing global repo layout or symlink policy outside purge.
🔵 P3 — Keep the exhaustive limiter-sweep regression test aligned with write_rate_limiter
📍 Where: crates/gitlawb-node/src/main.rs — sweep_evicts_expired_keys_from_every_limiter.
💥 What fails: The test’s limiters array still lists ten limiters and omits write_rate_limiter, while AppState::cleanup_rate_limiters sweeps it (H2). Removing write cleanup from production would not fail this test.
📜 Stated contract:
Comment above the test: "Every per-key limiter the router mounts must be swept"; companion
cleanup_rate_limiters_reaps_the_write_limiterinstate.rsdocuments H2.
🏷️ Attribution: PR-introduced omission when write_rate_limiter was added.
🔧 Required correction: Add write_rate_limiter to the test seed list and limiters closure (mirror the state-level guard).
🛠️ Author fix: One-line-array update plus assert loop — include in the same commit as any rate-limit touch if you are already editing main.rs tests.
🚫 Out of scope: Changing sweep interval or limiter algorithms.
- try_lock_repo_on routes the acquire through LockProbe so cancellation cannot return a lock-owning connection to the pool - run_pending_migrations runs on the connection holding the advisory lock, so a max_connections=1 purge cannot deadlock at startup - purge-spam canonicalizes repos_dir once; the ./repos default's CurDir component made validated_repo_disk_path reject every candidate - delete_archive fences on the generation verified empty; a PUT landing after the check makes the delete refuse instead of erasing it - refresh and archive delete run under transfer_bound while the repo lock is held, so a stalled transfer cannot pin the lock - init_handle returns the archive upload JoinHandle; create_repo awaits it before releasing the creation lock so a purge or retry cannot find the row with the PUT still in flight - ForkCloneGuard is created before the clone and claim_fork_disk_path creates the owner dir for the stamp; every clone failure, timeout, and cancellation path now cleans only what this attempt owns
- /graphql/ws serves a schema with no mutation root, so the socket can no longer execute task mutations outside the write-brake bucket; queries and subscriptions still run, and the schema carries the same Db, broadcast, and cursor-key data the resolvers consume - purge-spam skips a repo that still has unsettled receive-pack requests or pending ref transitions (checked under the repo lock, fail-closed) instead of stranding that work mid-effect
|
The coderabbit round is dispositioned on Fixed and resolved:
Left open, all one decision: the slug-normalization cluster (canonical-vs-mirror lock keys, full-DID cascade predicates, bounty serialization, sibling-survives policy) plus cache-miss publish fencing. These want a canonical storage/matching form decided once rather than patched predicate-by-predicate in this diff, so they are tracked as a follow-up. Declined with reasons on the threads: GraphQL reads sharing the write budget (documented tradeoff), trusted-origin gating (no such policy exists), and proxy-port hardening (deployment concern across every limiter). |
jatmn
left a comment
There was a problem hiding this comment.
I found issues that need to be addressed before this is ready.
Merge readiness
- GitHub reports
mergeable: CONFLICTING/mergeStateStatus: DIRTYon head96ed2b24ff4fe04679bccde2bd4d7dbf5fca020a. Rebase onto currentmain, resolve conflicts, and re-run review on the resolved diff. - The branch is at least three commits behind
origin/mainrelative to merge-base4fc041a3995965e199c5dc49d276717b40ab111a; security-sensitive listing/rate-limit work onmainmay not be present on this head until rebased. - CI signal is thin: only CodeRabbit (skipped) on the latest check rollup. Treat merge as blocked until repo-required checks run green on the rebased head.
Findings
🟠 P1 — close_issue documents durable-upload rollback but still uses plain release
📍 Where: crates/gitlawb-node/src/api/issues.rs (close_issue).
💥 What fails: The handler closes the issue on disk, then calls guard.release(close_result.is_ok()). If the object-store publish fails, the client gets a 5xx while the local repo still shows the issue closed—same gap create_issue already closes with release_compensating.
// added in this PR — promises rollback; no snapshot or compensator follows
// Snapshot the issue ref's pre-close OID so a failed durable upload can roll
// the close back on local disk, mirroring create_issue. ...
let close_result = git_issues::close_issue(&disk_path, &issue_id);
guard.release(close_result.is_ok()).await.into_result()?;🔎 Root cause: This diff adds operator-facing text claiming rollback “mirroring create_issue” without wiring the compensating release path that create_issue already uses.
📜 Stated contract:
In-diff comment at
close_issue(this PR) requires pre-close snapshot and local rollback on failed durable upload, mirroringcreate_issue.
create_issueusesrelease_compensating+git_issues::delete_issue_refon publish failure.
🏷️ Attribution: Incomplete claimed fix (comment introduced on this head; underlying plain release matches merge-base, but the new comment overclaims).
📌 In this PR:
close_issuecomment block and release call inapi/issues.rs
🔒 Unchanged on main: Tigris publish / RepoWriteGuard core; merge_pr plain release (not in this diff—do not conscript unless you re-land branch rollback work deliberately).
🔧 Required correction: Re-land the compensating pattern (snapshot pre-close ref, release_compensating rollback under the advisory lock, regression test) or remove the rollback comment so code and docs match. Mirror the existing create_issue compensator shape; do not redesign upload core.
🛠️ Author fix: Close the root cause on every listed in-diff row in one pass—comment, release path, and test together. Do not patch only the comment.
🚫 Out of scope: Rewriting merge_pr, receive-pack, or unrelated main call sites unless you explicitly re-import the lost branch rollback commit.
🟡 P2 — Purge execute recheck omits the symlink fail-closed gate used at selection
📍 Where: crates/gitlawb-node/src/admin.rs — local_refs_on_disk vs ref_count_on_disk, execute loop authoritative recheck.
💥 What fails: Selection rejects a symlink at the repo path (fail-closed). Between selection and --execute, a symlink can point at an empty bare repo outside repos_dir. ref_count_on_disk follows the link, reports zero refs, and the row is deleted; path_within may refuse remove_dir_all, producing DB/disk drift.
🔎 Root cause: This PR hardens selection with symlink_metadata but the execute-time authoritative recheck uses ref_count_on_disk, which skips the same symlink gate.
📜 Stated contract:
local_refs_on_disktreats a symlink at the target as non-empty (fail-closed) and documents that a symlink to an outside empty bare repo must never become a candidate.
🏷️ Attribution: PR-introduced asymmetry (new execute recheck path in admin.rs).
📌 In this PR:
ref_count_on_diskand execute-loop recheck under the per-repo lock inadmin.rs
🔒 Unchanged on main: path_within containment guard; selection tests already cover symlink at selection time.
🔧 Required correction: Share the selection symlink/metadata rules in ref_count_on_disk (or call a single helper) so execute-time emptiness matches selection fail-closed behavior. Add or extend a test that fails if execute recheck follows a symlink.
🛠️ Author fix: Fix the helper and execute path in one pass; do not tighten only dry-run selection.
🚫 Out of scope: Changing EXCLUDED_DIDS / target DID constants; broader purge policy.
🔵 P3 — build_ws_schema rustdoc contradicts the wired schema
📍 Where: crates/gitlawb-node/src/graphql/mod.rs (build_ws_schema).
💥 What fails: New rustdoc says “no query or mutation root exists on this type,” but the function builds QueryRoot + EmptyMutation + SubscriptionRoot. Maintainers can underestimate WS query load (mutations are correctly blocked; queries are not).
🔎 Root cause: In-diff documentation added when splitting WS schema does not match the actual Schema::build call.
📜 Stated contract:
In-diff
build_ws_schemadoc (this PR) claims the WS schema cannot execute queries or mutations.
🏷️ Attribution: PR-introduced misleading operator/maintainer copy.
📌 In this PR:
build_ws_schemamodule docs ingraphql/mod.rs
🔒 Unchanged on main: HTTP /graphql write brake; intentional EmptyMutation on WS.
🔧 Required correction: Rewrite the rustdoc to state mutations are disabled on WS but queries still run via QueryRoot, matching GitlawbWsSchema and server comments about subscriptions-only mutations.
🛠️ Author fix: Update the in-diff doc string in the same pass; do not re-expose MutationRoot on WS unless product intent changes.
🚫 Out of scope: Adding a write brake to /graphql/ws (documented as intentional in .env.example and tests).
…-coverage # Conflicts: # crates/gitlawb-node/src/main.rs
…t ws doc - close_issue snapshots the pre-close JSON and uses release_compensating, so a failed durable upload restores the ref instead of leaving the local repo closed while the client saw a 5xx (mirroring create_issue) - ref_count_on_disk rejects a symlink at the repo path, matching the selection-side fail-closed rule so the execute recheck cannot follow a link to an empty bare repo outside repos_dir - build_ws_schema rustdoc now states mutations are disabled via EmptyMutation while queries still run on QueryRoot
|
All findings are addressed on e9e86a1, on top of the origin/main merge at 082a37f. Merge readiness: the CONFLICTING state was the main.rs-to-lib.rs split. The branch's entire main.rs delta is re-applied into lib.rs: mod admin, run_admin_command and the admin dispatch, the graphql_ws_schema build and AppState wiring, the build_ip_limiter refactor with the new write_rate_limiter bucket, the cleanup_rate_limiters sweep call sites, and the rate_limit_env_tests module. test_harness.rs builds the ws schema and the write limiter too. Merge state should now be clean. P1 (close_issue): the compensating pattern is wired, matching create_issue's shape. The handler snapshots the pre-close JSON via get_issue, then calls release_compensating; on a failed durable upload the rollback rewrites the ref to the prior content via create_issue rather than leaving the local repo closed behind a 5xx. Plain release remains only for the close-failed or snapshot-unavailable arms. P2 (symlink recheck): ref_count_on_disk now refuses a symlink at the repo path (returns non-empty), the same fail-closed rule local_refs_on_disk applies at selection, so the execute-time recheck cannot follow a link to an empty bare repo outside repos_dir. P3 (ws doc): build_ws_schema rustdoc corrected: mutations are disabled via EmptyMutation while queries still run on QueryRoot (reads that ride the TaskReadBrake), so subscriptions-only refers to the write surface. Pre-push gate ran green: fmt --check, workspace clippy -D warnings, and the full deny-harness suite (53/53). |
jatmn
left a comment
There was a problem hiding this comment.
PR Review — Twigpine/node #196
feat(node): per-IP write-surface rate brake + purge-spam admin tool
Head: e9e86a15d41bddc9b952dbe2cbd2057c22fb0796 · Base: 55ca89bfffa2c81b7f7b6f30cee7743f000197ce
Merge readiness
- Required check
test (beta)failed on run 43114804801:ipfs_pin::tests::discovery_record_elapsed_leaves_neither_rowpanicked. That test lives inipfs_pin.rs, which is not in this PR’s 21-file diff;test (stable)passed on the same run. Treat as likely unrelated / flaky on beta until reproduced on a change that touchesipfs_pin, but merge state is BLOCKED until CI is green or maintainers waive beta. - Branch is up to date with
origin/main(0 behind, 69 ahead). No rebase conflict risk observed. - Prior
jatmnreview threads requested write-brake coverage and purge safety; current head adds broad route tests andadmin.rsguards. Remaining gaps are captured in Findings below (not a blanket re-request of older themes).
Findings
🟠 P1 — Purge execute can delete the DB row but leave the object-store archive
📍 Where: crates/gitlawb-node/src/admin.rs — run_purge_spam execute loop, after delete_repo_by_id succeeds.
💥 What fails: On the Ok(n) branch, if validated_repo_disk_path returns Err, the code increments disk_failed and continues before delete_archive runs, even though the DB row was already removed.
🔎 Root cause: Archive deletion is sequenced after disk-path validation; a continue short-circuits the whole tail of the success arm, including object-store cleanup.
📜 Stated contract:
Delete the object-store archive too… else it survives and can be downloaded into a later repo created with the same owner/name.
🏷️ Attribution: PR-introduced (admin.rs is new).
📌 In this PR:
run_purge_spamexecute path — archive delete skipped when post-delete path validation fails
🔒 Unchanged on main: N/A (new tool).
🔧 Required correction: After a successful delete_repo_by_id, still attempt delete_archive (with the same generation fence) when an object store is configured, even if local disk removal is refused; or reorder so path validation happens before the DB delete. Do not report a clean delete count when the archive was not addressed.
🛠️ Author fix: Close the purge execute lifecycle on every success arm in one pass: DB row, disk dir (when safe), and archive delete/share the same accountability rules. Adjust tests so a post-delete path-validation failure still exercises archive cleanup (or proves intentional skip with an explicit summary counter).
🚫 Out of scope: Redesigning Tigris delete semantics on main.
🟡 P2 — Purge execute summary under-reports unsettled-work skips
📍 Where: crates/gitlawb-node/src/admin.rs — PurgeSummary and final println! after --execute.
💥 What fails:
skipped_unsettledis incremented whenhas_unsettled_repo_workis true, but the closing summary line never prints that counter.- When
has_unsettled_repo_workreturnsErr, the repo is skipped fail-closed but no summary field is incremented (unlikeskipped_store_error/skipped_locked).
🔎 Root cause: Operator-facing accounting was not kept in sync with the new PurgeSummary fields and the unsettled-work guard.
📜 Stated contract:
skipped_unsettled— Candidates skipped because receive-pack or transition work still has in-flight work on the repo.
🏷️ Attribution: PR-introduced.
📌 In this PR:
PurgeSummarystruct- execute-loop
println! has_unsettled_repo_workerror arm
🔒 Unchanged on main: N/A.
🔧 Required correction: Include skipped_unsettled in the final execute summary (and dry-run guidance if applicable). Add a dedicated counter (or reuse an existing skip bucket) for unsettled probe failures, with a regression test that forces has_unsettled_repo_work Err and asserts the summary reflects it.
🛠️ Author fix: Align every skip/fail path in the execute loop with a PurgeSummary field and the printed totals in one change.
🚫 Out of scope: Changing unsettled-work SQL semantics.
🟡 P2 — close_issue compensation does not honor git_service_timeout like create_issue
📍 Where: crates/gitlawb-node/src/api/issues.rs — close_issue compensating branch (new hunk).
💥 What fails: On successful local close, if durable upload fails, compensation calls git_issues::create_issue(path, …) with no deadline, while the sibling create_issue handler passes git_bin and a git_service_timeout_secs deadline into delete_issue_ref inside release_compensating.
🔎 Root cause: The new close path discards git_bin / deadline (let _ = (git_bin, deadline)) even though the in-diff comment claims mirroring create_issue.
📜 Stated contract:
Snapshot the issue's pre-close JSON so a failed durable upload can roll the close back on local disk, mirroring create_issue.
🏷️ Attribution: PR-introduced.
📌 In this PR:
close_issuecompensating closure
🔒 Unchanged on main: create_issue compensation pattern (use as reference).
🔧 Required correction: Apply the same timeout-bound git rewrite strategy create_issue uses (or an equivalent bounded compensation) when restoring prior_json after upload failure.
🛠️ Author fix: Make close and create compensation symmetric in one pass; add/extend a test that upload failure on close triggers bounded rollback behavior.
🚫 Out of scope: Rewriting release_compensating for all handlers.
🟡 P2 — create_repo can return 201 while the initial archive upload failed
📍 Where: crates/gitlawb-node/src/api/repos.rs — create_repo (new advisory-lock + create_upload sequence).
💥 What fails: After db.create_repo, the handler awaits the background upload with let _ = upload.await and returns 201 Created regardless of join/upload failure. Comments assert the row, disk, and archive are consistent before the lock is released.
🔎 Root cause: Upload outcome is not propagated to the HTTP response; error paths can also drop the creation lock before the upload task finishes (success path only awaits upload).
📜 Stated contract:
Keep the initial archive upload inside the creation lock… Row + on-disk dir now both exist consistently; the race window is closed.
🏷️ Attribution: PR-introduced.
📌 In this PR:
create_reposuccess and error paths aroundcreate_upload/repo_lock.release()
🔒 Unchanged on main: Prior create flow without this lock/upload ordering.
🔧 Required correction: Treat a failed create-only upload like other archive failures relevant to creation (fail the request or compensating-delete row+disk), and keep the advisory lock until the upload task completes on all exit paths that inserted a row, matching the stated race comment.
🛠️ Author fix: Close create/upload/lock lifecycle on every exit path in one pass; add regression coverage for upload failure during create under Tigris.
🚫 Out of scope: Changing fork/upload semantics outside create_repo.
🔵 P3 — purge-spam cannot take GITLAWB_TIGRIS_BUCKET from CLI after the subcommand
📍 Where: crates/gitlawb-node/src/config.rs (tigris_bucket not global = true); lib.rs run_admin_command.
💥 What fails: gitlawb-node purge-spam --tigris-bucket … is rejected by clap; only repos_dir and database_url are global. Tigris-backed authoritative purge requires env or flags before purge-spam.
🔎 Root cause: Admin wiring reads config.tigris_bucket, but the CLI surface for purge does not expose it post-subcommand.
📜 Stated contract:
Build the object store so purge is authoritative against remote state… When no bucket is configured this stays None and purge is local-only.
🏷️ Attribution: PR-introduced admin entrypoint.
📌 In this PR:
PurgeSpamsubcommandrun_admin_commandobject-store setup
🔒 Unchanged on main: Daemon flag layout.
🔧 Required correction: Mark tigris_bucket (and any other store fields purge needs) global = true, or document in purge-spam --help / operator runbooks that bucket config must come from env or pre-subcommand flags.
🛠️ Author fix: Pick one operator path and make help, .env.example, and clap agree in one pass.
🚫 Out of scope: New object-store providers.
- purge-spam execute: the object-store archive delete now runs on every successful row delete; a refused or failed local dir removal no longer strands the archive (P1) - purge-spam summary: skipped_unsettled is now printed, and a dedicated skipped_unsettled_probe counter records fail-closed skips when the unsettled-work probe itself errors (P2) - close_issue: rollback compensation uses create_issue_bounded, the same deadline-bounded git strategy as create_issue's compensator, so a hung git cannot stall the write guard past git_service_timeout (P2) - create_repo: the initial archive upload's outcome now gates the 201; a failed or fenced upload rolls back the row and disk dir and surfaces an error instead of reporting success (P2) - purge-spam: tigris_bucket is clap-global so --tigris-bucket parses after the subcommand (P3)
|
All five findings are addressed on 1187dca. P1 (archive orphan): the execute loop no longer continues past delete_archive on a disk-path validation failure. The Ok(n) arm now treats disk removal as best-effort and always runs the archive delete (same generation fence), so a refused or failed local removal cannot strand the object-store copy of a row that was just deleted. P2 (unsettled accounting): skipped_unsettled is now in the final summary line, and a dedicated skipped_unsettled_probe counter records the fail-closed skip when has_unsettled_repo_work itself errors, so an operator can distinguish work in flight from could-not-check. P2 (bounded close rollback): git/issues.rs gained create_issue_bounded, which runs the same hash-object plus update-ref write through run_bounded_git with the caller deadline and git_bin. close_issue compensator calls it, matching create_issue deadline strategy; the discard of git_bin and deadline is gone. P2 (create_repo upload): init_handle upload task now returns a Result, and create_repo awaits it. An upload failure, a precondition-lost fence (the archive key then belongs to another writer), or a task panic rolls back the create: the row is deleted via delete_repo_by_id, the disk dir removed, the lock released, and the client sees an error instead of 201. P3 (tigris flag): tigris_bucket is now clap-global, so gitlawb-node purge-spam --tigris-bucket parses post-subcommand. Pre-subcommand and env paths still work. Pre-push gate ran green: fmt --check, workspace clippy -D warnings, deny_harness 53/53. The deferred slug-normalization and cache-miss threads remain open as the tracked follow-up set. |
jatmn
left a comment
There was a problem hiding this comment.
I found issues that need to be addressed before this is ready.
Merge readiness
- All required checks are green on head
1187dcab79c93fb38188c50138d30db65f00e265(stable + beta tests, fmt/clippy, build). - Branch is 0 commits behind
origin/main(merge-base55ca89bfffa2c81b7f7b6f30cee7743f000197ce); mergeable. - GitHub review decision is still
CHANGES_REQUESTEDfrom earlier heads; several prior jatmn items (close_issue compensating release, purge archive/symlink siblings) are fixed on this head — see findings below for what remains.
Findings
🟠 P1 — Re-land merge durable-upload rollback claimed in release notes (close path fixed; merge path not)
📍 Where: crates/gitlawb-node/src/api/pulls.rs (merge_pr).
💥 What fails: store::merge_branch mutates the bare repo, then guard.release(merge_result.is_ok()) publishes to durable storage. On definite publish refusal the handler returns 5xx, but there is no git rollback of the merge commit (unlike close_issue, which now uses release_compensating with a pre-close snapshot in api/issues.rs).
guard.release(merge_result.is_ok()).await.into_result()?;
let merge_sha = merge_result.map_err(|e| AppError::Git(e.to_string()))?;
state.db.merge_pr(&pr.id, &merger_did).await?;🔎 Root cause: This PR advertises improved close/merge behavior on storage failure and implements compensation for close, but merge_pr still uses plain release with no compensating closure.
📜 Stated contract:
CodeRabbit release notes on this PR: "Improved pull-request merge handling when durable storage operations fail."
🏷️ Attribution: Incomplete claimed fix on this branch (close fixed; merge not). Underlying merge-without-compensation also exists at merge-base — not a mandate to refactor all main writers.
📌 In this PR:
merge_prhandler (pulls.rs)- any merge rollback helpers/tests this branch previously added for
merge_prbut did not keep on head
🔒 Unchanged on main: RepoWriteGuard publish/quarantine core, receive_pack path, Tigris conditional PUT semantics.
🔧 Required correction: Mirror the close_issue / create_issue pattern for merge_pr: capture pre-merge refs (or merge parent), use release_compensating so definite publish refusal rewinds the local merge while the advisory lock is held, and add a RED test analogous to create_issue_rolls_back_local_ref_when_publish_refuses.
🛠️ Author fix: Land merge compensation, helpers, and tests together in one pass. Do not patch only release notes or only comments.
🚫 Out of scope: Rebuilding every other write handler on main that never compensated.
🟡 P2 — create_repo upload rollback overclaims when the DB row delete fails
📍 Where: crates/gitlawb-node/src/api/repos.rs (create_repo, initial archive upload inside the creation lock).
💥 What fails: When the empty-repo upload fails, the handler calls delete_repo_by_id, then remove_dir_all, then returns an error stating the create was rolled back. If delete_repo_by_id errors (warn-only), execution still removes the on-disk directory and returns the rollback wording.
🔎 Root cause: Rollback success is not gated on both row delete and dir removal succeeding; operator-visible text overstates recovery.
📜 Stated contract:
In-diff comment above the upload block: a failed upload must "roll the create back (row + dir)" before surfacing the error instead of reporting success.
🏷️ Attribution: PR-introduced (new upload-in-lock + rollback block on this branch).
📌 In this PR:
create_repoupload failure path only
🔒 Unchanged on main: Proof consume ordering, iCaptcha core, unrelated handlers.
🔧 Required correction: If delete_repo_by_id fails, do not claim full rollback — return an error that states partial state (orphan row and/or orphan dir) and/or retry row delete before removing disk; add a regression test where mocked delete_repo_by_id fails.
🛠️ Author fix: Close the handler message and control flow together; do not change only the user-facing string.
🚫 Out of scope: Redesigning repo creation or Tigris upload pipeline.
🟡 P2 — Purge execute: post-commit window allows mirror ingest before remove_dir_all
📍 Where: crates/gitlawb-node/src/admin.rs (run_purge_spam execute loop); interacts with crates/gitlawb-node/src/db/mod.rs (delete_repo_by_id slug xact lock) and upsert_mirror_repo.
💥 What fails: Purge holds try_lock_repo (SHA-based session advisory lock) across delete + disk removal. delete_repo_by_id takes the slug hashtextextended transaction lock only until commit. After the row delete commits, a concurrent upsert_mirror_repo can insert a new repos row for the same normalized slug while purge still runs remove_dir_all on that owner/name path — wiping disk under a live row.
🔎 Root cause: Slug-level serialization ends at transaction commit; on-disk deletion is not serialized against mirror ingest the way the in-transaction sibling check is.
📜 Stated contract:
delete_repo_by_idcomment: slug xact lock serializes purge against concurrentupsert_mirror_repo; purge module goal that DB row and on-disk dir stay consistent.
🏷️ Attribution: PR-introduced (admin.rs new; expanded cascade in this diff). DB test purge_serializes_with_mirror_ingest_via_advisory_lock holds a session slug lock for the whole scenario — live purge does not hold that lock across disk delete.
📌 In this PR:
run_purge_spamexecute loop (after successfuldelete_repo_by_id, before / during disk removal)- add execute-path regression mirroring the DB race test with the real purge lock sequence
🔒 Unchanged on main: Global mirror ingest API; do not remove slug lock from upsert_mirror_repo.
🔧 Required correction: Hold slug advisory serialization from before delete through disk removal, or re-check sibling_exists / row absence immediately before remove_dir_all and abort disk delete if a new row landed.
🛠️ Author fix: Close purge execute + test in one pass.
🚫 Out of scope: Transactional purge across DB+disk+S3 in one atomic step.
🔵 P3 — Sweep integration test still omits write_rate_limiter
📍 Where: crates/gitlawb-node/src/lib.rs — sweep_evicts_expired_keys_from_every_limiter.
💥 What fails: The test seeds and asserts cleanup for ten limiters but not write_rate_limiter, even though AppState::cleanup_rate_limiters sweeps it (H2; guarded separately in state.rs).
🔎 Root cause: New limiter added in this PR without extending the cross-limiter regression array.
📜 Stated contract:
Comment above the test: "Every per-key limiter the router mounts must be swept."
🏷️ Attribution: PR-introduced omission.
📌 In this PR:
sweep_evicts_expired_keys_from_every_limiterlimiter list only
🔒 Unchanged on main: Sweep interval algorithms.
🔧 Required correction: Add state.write_rate_limiter to the seed loop and limiters closure (same pattern as cleanup_rate_limiters_reaps_the_write_limiter).
🛠️ Author fix: One-line array update in the same commit if touching limiter tests.
🚫 Out of scope: Changing production cleanup wiring (already correct in state.rs).
- merge_pr snapshots the pre-merge target tip and uses release_compensating to rewind it via bounded update-ref on a refused publish, matching the close_issue pattern; regression proves the ref returns to its prior OID after a refused upload (P1) - create_repo rollback names residual state instead of claiming a full rollback when the row delete fails or dir removal fails (P2) - purge execute re-checks row absence via get_repo_by_id immediately before remove_dir_all and refuses the disk delete when a row reappeared after commit, closing the post-commit mirror-ingest window (P2) - sweep_evicts_expired_keys_from_every_limiter now seeds and overrides the write_rate_limiter TTL (P3)
|
All four findings are addressed on 261f7ab. P1 (merge rollback): merge_pr now snapshots the target ref pre-merge OID via read_ref_bounded and uses release_compensating, whose closure rewinds refs/heads/ through run_bounded_git inside git_service_timeout, the same shape as close_issue. merge_pr_rewinds_local_merge_when_publish_refuses asserts rev-parse returns to the pre-merge tip after a refused publish. P2 (create_repo overclaim): the rollback now tracks row_deleted and dir_removed independently, and the returned error names the residual state (the repo row remains / the on-disk directory remains / both) with operator cleanup required, instead of claiming a full rollback when either step failed. P2 (purge post-commit window): the execute loop re-checks db.get_repo_by_id immediately before remove_dir_all and refuses the disk delete when a repo row reappeared after commit, counting disk_failed; a probe error fails closed. P3 (sweep omission): sweep_evicts_expired_keys_from_every_limiter now lists write_rate_limiter in the limiters closure and gives it the same RateLimiter::new(10, window) TTL override as the others; the omission was in both spots, which is why the first pass of the fix went red. Pre-push gate ran green: fmt --check, workspace clippy -D warnings, deny_harness 53/53. The deferred slug-normalization and cache-miss threads remain open as the tracked follow-up set. |
jatmn
left a comment
There was a problem hiding this comment.
I found issues that need to be addressed before this is ready.
Merge readiness
Required checks are red on this head. GitHub Actions run 43142788397 reports test (stable) and test (beta) failed on cargo test, while fmt + clippy, build --release, MSRV, and Docker smoke passed. The live target branch main at merge-base 55ca89b is green on the same workflow class. The PR verification section claims cargo test --workspace is green — that does not match the required checks on 261f7ab3. Reconcile the failing test names from CI logs before merge (this is merge hygiene on this head, not a mandate to fix unrelated main CI debt).
Base is current. Merge-base equals live main (55ca89b); no stale-base rebase is required for metadata drift.
🟠 P1 — Post-delete purge must probe slug liveness before disk or archive destruction
📍 Where: crates/gitlawb-node/src/admin.rs (run_purge_spam execute loop, post-delete_repo_by_id disk removal and archive delete).
💥 What fails: After the canonical row is deleted, the pre-remove_dir_all guard calls db.get_repo_by_id(&c.id) for the purged primary key. A concurrent upsert_mirror_repo can insert a new row for the same owner/name slug with id {owner_short}/{name} (db/mod.rs) while purge still holds the session try_lock_repo lock — upsert_mirror_repo does not take that session lock, only the transaction-scoped lock_repo_slug that ends when delete_repo_by_id commits.
The inline comment names this mirror-ingest window, but the probe cannot see the new row:
// concurrent mirror ingest can insert a new row for this slug before remove_dir_all runs.
let row_gone = match db.get_repo_by_id(&c.id).await {
Ok(None) => true,
// ...
};When row_gone is wrongly true, purge removes the on-disk bare repo. Archive delete still runs unconditionally afterward (delete_archive is not gated on slug absence), so object-store bytes can be removed while a live mirror repos row remains — DB / disk / archive drift and broken serve for that slug.
🔎 Root cause: Post-delete destructive steps use primary-key absence instead of slug-level liveness. The PR’s own comment describes slug-level re-check; the implementation checks only the deleted id.
📜 Stated contract:
Re-check immediately before the destructive op and abort the disk delete when a fresh row landed rather than wipe disk under a live row.
🏷️ Attribution: PR-introduced. There is no crates/gitlawb-node/src/admin.rs at merge-base 55ca89b; this execute path and guard are new in this diff.
📌 In this PR:
run_purge_spampost-delete disk guard (get_repo_by_id)- unconditional
delete_archiveafter row delete - operator dry-run / execute summary that can report disk+archive removal under a surviving slug row
🔒 Unchanged on main: In-transaction lock_repo_slug serialization between delete_repo_by_id and upsert_mirror_repo (existing DB test purge_serializes_with_mirror_ingest_via_advisory_lock). This finding is only the post-commit window after that xact lock is released.
🔧 Required correction: Before remove_dir_all and before delete_archive, treat the slug as live when db.get_repo(&c.owner_did, &c.name) (or equivalent owner-key-normalized lookup) returns any row. Skip both destructive steps when a row exists; count separately in PurgeSummary. Add a regression test racing upsert_mirror_repo after delete_repo_by_id commits but before disk removal (mirror id owner/name).
🛠️ Author fix: Close slug-level liveness on both disk and archive paths in one pass; do not patch only the get_repo_by_id call while leaving archive delete ungated.
🚫 Out of scope: Reworking peer-sync ingest to take session repo locks; changing mirror id format; fixing pre-existing ingest paths on unchanged main.
🟡 P2 — Issue write HTTP brake must not run before the INV-12 read gate
📍 Where: crates/gitlawb-node/src/server.rs (issue_write_routes + write_braked); crates/gitlawb-node/src/api/issues.rs (close_issue, create_issue_comment).
💥 What fails: issue_write_routes is wrapped with write_braked, so rate_limit_by_ip on the write bucket is the outermost middleware on:
POST /api/v1/repos/{owner}/{repo}/issues/{id}/closePOST /api/v1/repos/{owner}/{repo}/issues/{id}/comments
An exhausted write bucket returns 429 before authorize_repo_read runs in the handler. For callers probing private or missing repos, that breaks the documented status ordering: 429 can appear without the read gate having returned the opaque 404 used for hidden repos.
🔎 Root cause: This PR applied the shared HTTP write brake to the issue write router. The INV-12 ordering comment in close_issue (and the read-gated comment flow) predates this PR unchanged at 55ca89b; merge-base issue_write_routes had no .write_braked. The new outer middleware runs before the handler; the in-handler close_issue_rate_limiter still honors post-gate ordering but does not undo the outer brake.
📜 Stated contract:
Rate limiting runs AFTER this gate so a 429 cannot distinguish a hidden repo from a missing one (INV-12 read-denial status contract).
🏷️ Attribution: PR-activated. Merge-base issue_write_routes ended at add_auth_layers with no write_braked; head adds .write_braked(&write_ip_limiter).
📌 In this PR:
server.rsissue_write_routes.write_brakedclose_issue(read-gated close + innerclose_issue_rate_limiter)create_issue_comment(read-gated comment write)- new integration tests that assert write brake on issue comment without INV-12 ordering
🔒 Unchanged on main: Other write groups (merges, stars, hooks, etc.) that are not read-gated before mutation may keep outer write_braked; do not remove their middleware or extend INV-12 refactors to those routes in this PR.
🔧 Required correction: Debit the write bucket for issue close and issue comment inside each handler immediately after authorize_repo_read (same placement as close_issue_rate_limiter), or split issue close/comment onto a router without outer write_braked while still debiting write_rate_limiter post-gate. Add regression tests: under an exhausted write bucket, a caller denied read access still receives the read-gate status (404/403), not 429, for both routes.
🛠️ Author fix: Fix ordering on both issue write routes together; do not move only close_issue while leaving comment on the outer brake.
🚫 Out of scope: Changing INV-12 behavior on unrelated read routes; redesigning global write-brake routing for the whole server; pre-existing GITLAWB_CLOSE_ISSUE_RATE_LIMIT env parsing (unchanged at merge-base).
…he read gate (#196) Three items from review round 6: - merge_pr rollback test seeded commits with git commit-tree under no configured identity, which fails on CI runners. The helper now passes -c user.name/-c user.email per invocation. - run_purge_spam's post-delete guard probed the deleted primary key, but a concurrent upsert_mirror_repo lands a NEW row for the same slug with a mirror id, and the archive delete ran unconditionally. The probe now reads slug liveness via get_repo(owner, name) and gates BOTH the disk and archive deletes, counted as skipped_slug_live. - issue_write_routes carried the outer per-IP write brake, so an exhausted bucket answered 429 before authorize_repo_read could return the opaque 404 a hidden repo owes the caller. The brake is now debited inside close_issue and create_issue_comment, immediately after the read gate; the route group carries no outer brake.
|
All three items fixed in Merge readiness: P1, post-delete slug liveness: the guard now probes P2, issue write brake ordering: Verified locally: |
jatmn
left a comment
There was a problem hiding this comment.
I found issues that need to be addressed before this is ready.
Merge readiness
- Required GitHub checks
test (stable)andtest (beta)failed on head662b9a3a31cbc973d4a4f728fcb5ccfe74f3b3aa(workflow run 43164147039). fmt, clippy, build, MSRV, and other gates are green. - PR-owned CI blocker: Finding 1 below (
issue_comment_is_rate_limited_by_ip) fails locally and is in this diff; fixing or removing it is required for a greencargo teston this branch. - Pre-existing CI friction (not in
## Findings):test_support::tests::merge_pr_fails_closed_when_target_is_only_a_tagandmerge_pr_advances_local_branch_shadowed_by_same_named_tagcallgit tag releasewithout-mor-F; on dumb CI terminals that fails with “EDITOR unset.” The same invocation is present at merge-base55ca89bf— not introduced by this feature diff. It can still block required checks on this head until those helpers pass-m(or CI sets a tag message). That is merge hygiene, not a request to redesign merge semantics in this PR. - Out of this diff: If
test (beta)also reportsreconciliation::sweep_restores_tigris_cache_miss_and_skips_missing, that file is unchanged on this branch; treat as upstream flake or main-track follow-up, not a #196 code finding. - Branch is mergeable against current
origin/main(55ca89bfffa2c81b7f7b6f30cee7743f000197ce, 0 behind). No conflict markers in the prepared checkout. - Prior review threads on advisory-lock pinning, purge/mirror serialization, merge/close compensating upload refusal (merge path), archive delete after row delete, symlink recheck, and limiter sweep are addressed on this head per code inspection and your recent comments; not re-opened except where evidence below still fails.
- Drift check (withdrawn from draft): Outer
write_brakedon star/merge routes does not violate INV-12 the way issue routes do (exhausted bucket yields the same 429 before the handler for missing and real slugs). InvalidGITLAWB_*_RATE_LIMITparse → default with warn is intentional. Purge/mirror xact slug lock and related tests match your stated fixes.
🟠 P1 — Remove or rewrite the issue-comment router throttle test so it matches INV-12 wiring
📍 Where: crates/gitlawb-node/src/api/repos.rs — issue_comment_is_rate_limited_by_ip
💥 What fails: The test exhausts write_rate_limiter, POSTs to /api/v1/repos/someowner/somerepo/issues/1/comments without a signature, and expects 429. The router returns 401 because issue write routes deliberately omit the outer write_braked middleware; throttling runs inside create_issue_comment only after authorize_repo_read and authentication.
assertion `left == right` failed: issue-write routes must be IP-throttled by the write brake
left: 401
right: 429
🔎 Root cause: The test was added as a “representative REST write group” sibling to star/label routes that use write_braked, but issue comments follow the in-handler brake pattern documented for INV-12.
📜 Stated contract:
Issue write routes: an outer per-IP brake would answer 429 before
authorize_repo_readcould return the opaque 404 a hidden repo owes the caller (INV-12).
🏷️ Attribution: PR-introduced (this test is new in the diff).
📌 In this PR:
crates/gitlawb-node/src/api/repos.rs— deleteissue_comment_is_rate_limited_by_ip, or replace it with a test that signs/auths and asserts 429 from the handler path (mirrorissues.rsan_exhausted_write_bucket_cannot_preempt_the_read_gate/ existingcreate_issue_commentrate tests)
🔒 Unchanged on main: server.rs issue route group without outer brake; do not “fix” by attaching write_braked to issue routes.
🔧 Required correction: Drop the misleading router-only test or rewrite it to exercise in-handler debit with proper auth and read-gate ordering.
🛠️ Author fix: One pass on repos.rs tests only; keep coverage in issues.rs as the authority for issue-write throttle ordering.
🚫 Out of scope: Moving issue routes to middleware-level throttling.
🟡 P2 — Close-issue upload rollback must rewrite the resolved issue ref, not the URL prefix
📍 Where: crates/gitlawb-node/src/api/issues.rs — close_issue compensating release_compensating closure
💥 What fails: When the client closes via an 8-char prefix, git_issues::close_issue resolves to the full UUID and updates refs/gitlawb/issues/{full_id}. If durable upload then refuses, the compensator calls create_issue_bounded with the path issue_id (the prefix), creating/updating refs/gitlawb/issues/{prefix} instead of the canonical ref. The client sees 5xx while the full ref stays closed.
🔎 Root cause: Compensating rollback was added on this branch (merge-base used plain release on close with no upload compensation). The compensator copies the path segment into the git ref writer without using the same resolved id close_issue used for the mutation.
📜 Stated contract:
Handler comment: on failed upload, “rewrites the ref back to the pre-close content instead of leaving the local repo closed while the client saw 5xx.”
git/issues.rstest_close_issue_via_prefix: close-by-prefix mutates the resolved full ref.
🏷️ Attribution: PR-introduced (compensating close on close is new on this branch).
📌 In this PR:
crates/gitlawb-node/src/api/issues.rs— compensator id passed tocreate_issue_bounded- Add a regression test: close via prefix + refused publish → local ref content returns to pre-close on the full id
🔒 Unchanged on main: Tigris upload core; create_issue compensator pattern.
🔧 Required correction: Derive the resolved full issue id (from close_result or resolve_issue_id) and pass that into create_issue_bounded in the compensator.
🛠️ Author fix: Handler + one load-bearing test in the same commit; do not change unrelated write paths.
🚫 Out of scope: Prefix resolution behavior in git/issues.rs itself; compensating every other write path on main that never had rollback at merge-base.
…test, tag fixtures get -m (#196) - close_issue's compensating rollback now rewrites the resolved full issue id; a close-by-prefix refused at publish left the full ref closed while the client saw 5xx (P2, regression test included) - issue_comment_is_rate_limited_by_ip asserted an outer 429 the issue write routes deliberately lack (P1); issues.rs owns the ordering - git tag fixtures pass -m so dumb CI terminals do not hit "EDITOR unset"
|
All three items are on
On The earlier coderabbit deferrals are now tracked in #583 and those threads are resolved. |
jatmn
left a comment
There was a problem hiding this comment.
I found issues that need to be addressed before this is ready.
Merge readiness
test (stable) failed on the latest workflow run (job log); test (beta) passed. Treat the stable failure as a merge blocker until the failing case is confirmed PR-owned or the run is green on head.
GitHub still reports mergeStateStatus: BLOCKED. fmt/clippy, build --release, MSRV, cargo audit, and Rust CodeQL passed on that run.
The PR head (43d4f59f) matches the fetched origin/main merge-base (55ca89bf) — no stale-branch conflict surfaced in this checkout.
Findings
🟡 P2 — Align in-diff operator copy with every surface that debits GITLAWB_WRITE_RATE_LIMIT
📍 Where: .env.example (new write-rate section), crates/gitlawb-node/src/lib.rs comment above write_rate_limiter, crates/gitlawb-node/src/state.rs write_rate_limiter field rustdoc, and the PR description consumer list (same omissions as .env.example).
💥 What fails: An operator sets GITLAWB_WRITE_RATE_LIMIT from the new template (or reads the PR body) and treats the bullet list as the full per-IP aggregate. On head, these also debit the shared write bucket but are missing or understated in that copy:
POST …/issues/{id}/close— handler-level debit ofwrite_rate_limiternew in this PR (merge-base only usedclose_issue_rate_limiter), plus the separateGITLAWB_CLOSE_ISSUE_RATE_LIMITPOST …/issues/{id}/comments— handler-levelwrite_rate_limiternew in this PR- Webhook
POST/DELETEonwrite_routes— routes existed at merge-base but were unbraked; this PR puts them behindwrite_braked GET …/visibilityon the samewrite_brakedgroup — intentional in code (harmless read debiting write bucket) but not called out in the new env textstate.rsrustdoc still says GraphQLMutationRootonly, while HTTP/graphqldebits queries and playground GET as well (.env.exampledocuments HTTP; field doc does not)
This is operator-contract drift on surfaces this PR newly attached to the bucket, not a request to fix pre-existing unbraked behavior on main.
🔎 Root cause: The write brake landed with a new env section and rustdocs that repeat an incomplete enumeration; the implementation and server.rs wiring are broader than the listed consumers.
📜 Stated contract:
Max non-creation write requests per client IP per hour: issue/PR comments, labels, stars, merges, protect/unprotect, replicas, visibility, tasks, bounties, profile, and all GraphQL HTTP requests.
— new .env.example section on this branch (presents itself as the consumer inventory for GITLAWB_WRITE_RATE_LIMIT)
🏷️ Attribution: PR-introduced copy drift (new env block; new write_rate_limiter debits on issue paths; webhooks newly behind write_braked). Not a pre-existing main doc bug — merge-base had no GITLAWB_WRITE_RATE_LIMIT section and did not debit the write bucket on close_issue.
📌 In this PR:
.env.examplewrite-rate sectionlib.rscomment abovebuild_ip_limiter("GITLAWB_WRITE_RATE_LIMIT", …)state.rswrite_rate_limiterfield documentation- PR description “What” bullet list (if still posted as operator guidance)
- Short note: issue close/comment use handler debits (INV-12), not outer
write_braked
🔒 Unchanged on main: Bucket limits, creation/push/peer buckets, and route behavior outside this diff.
🔧 Required correction: Rewrite every in-diff operator surface to the same complete consumer list for head: add issue close, issue comments, webhooks, dual-bucket note for close, visibility GET on the write group, and HTTP GraphQL (queries + playground GET, not only mutations). Align state.rs rustdoc with .env.example on GraphQL HTTP vs WebSocket.
🛠️ Author fix: Update all listed surfaces in one pass so no sibling comment or env block still advertises the short list.
🚫 Out of scope: Retuning defaults, new env vars, or moving issue routes to router middleware (INV-12).
🟡 P2 — close_issue must not consume write_rate_limiter when only the close cap rejects
📍 Where: crates/gitlawb-node/src/api/issues.rs — close_issue; crates/gitlawb-node/src/rate_limit.rs — check / is_throttled.
💥 What fails: After the read gate:
if !state.write_rate_limiter.check(&key).await
|| !state.close_issue_rate_limiter.check(&key).awaitcheck records a timestamp on allow. If the write bucket has capacity but GITLAWB_CLOSE_ISSUE_RATE_LIMIT is exhausted, the handler returns 429 after consuming one write slot. The IP loses shared write quota (stars, comments, GraphQL HTTP, etc.) without completing a close.
At merge-base, close_issue only called close_issue_rate_limiter.check — this cross-bucket bleed does not exist on main.
🔎 Root cause: This PR added write_rate_limiter to close_issue but chains two consuming checks with ||, so the first successful check always records even when the second rejects.
📜 Stated contract:
push traffic must keep its own bucket after close_issue is exhausted
— close_issue_rate_limit_does_not_drain_push_bucket in issues.rs (existing test that encodes cross-bucket isolation for close vs push; same product rule applies to close vs the shared write bucket)
📜 Stated contract (mechanism):
Unlike [
check], it records nothing and never inserts a new key … WITHOUT perturbing the per-unit budget the consumingcheckmaintains.
— RateLimiter::is_throttled in rate_limit.rs
🏷️ Attribution: PR-introduced (write_rate_limiter.check added in this diff only). Pre-existing close-only limiting on main is not the defect; the new write debit is.
📌 In this PR:
close_issuehandler debit logic (afterauthorize_repo_read, before auth work)- New regression test: exhaust close bucket while write bucket has capacity; assert write counter/timestamp unchanged (mirror
close_issue_rate_limit_does_not_drain_push_bucket)
🔒 Unchanged on main: INV-12 ordering (read gate before 429); issue routes must stay off outer write_braked.
🔧 Required correction: Use non-consuming is_throttled on both buckets to decide rejection; only call consuming check on both when the request will proceed past the rate gates (or equivalent logic with no partial debit). Add the regression test above.
🛠️ Author fix: Fix dual-bucket behavior and test in the same commit; do not merge the two limiters or add router middleware on issue routes.
🚫 Out of scope: Changing GITLAWB_CLOSE_ISSUE_RATE_LIMIT parsing (unchanged vs base), other write routes, or framework-wide rate-limit redesign.
… complete write-brake operator copy (#196) close_issue chained two consuming limiter checks, so a close refused by its own cap still debited the shared write bucket. Probe both with is_throttled first and consume only when both have room. Rewrite the write-brake consumer list in .env.example, lib.rs and state.rs to cover every surface on the bucket.
|
Both P2s are addressed on 4b483a3.
The
The failing case on the previous head was |
jatmn
left a comment
There was a problem hiding this comment.
I rechecked head 4b483a35 after your Nov 12 fixes. The prior P1 block (close/merge compensating publish, purge archive/disk/slug-live, close dual-bucket probes, sweep test) looks landed; deny harness 53/53 passes locally. Two low-severity items remain on this PR’s write-brake work.
Merge readiness
- GitHub still shows CHANGES_REQUESTED on this PR until a fresh approving review lands on
4b483a35. - Required Rust jobs on the latest run are green (fmt, clippy, MSRV, release build, deny-related unit surface);
test (stable)/test (beta)were still pending at review time — confirm they finish green on4b483a35(you noted an out-of-diffreconciliationflake on the prior head). - Branch is mergeable with current
main; no conflict blocker observed. - Slug-normalization / mirror-ingest follow-ups you deferred to a dedicated issue are acknowledged and are not re-listed here as PR-scope defects.
Findings
🔵 P3 — close_issue can still debit write_rate_limiter without closing when both buckets look open
📍 Where: crates/gitlawb-node/src/api/issues.rs (close_issue, ~298–306)
💥 What fails: After both is_throttled probes pass, write_rate_limiter.check runs before close_issue_rate_limiter.check. If the write check succeeds and the close check then fails (close already at cap due to a concurrent close), the write slot is consumed but no close runs.
🔎 Root cause: Sequential consuming checks after non-consuming probes; the second failure does not roll back the first debit.
📜 Stated contract:
In-handler comment: a close refused by either cap spends nothing from the other bucket.
🏷️ Attribution: PR-introduced (dual-bucket + probe pattern on this branch).
📌 In this PR:
close_issuerate-limit gate only
🔒 Unchanged on main: Outer write_braked placement on other routes.
🔧 Required correction: Debit both buckets only when both checks will succeed (e.g. paired non-consuming admission then paired check, or restore write slot if close check fails), and extend tests beyond the fully-saturated-bucket case to cover write-ok/close-fail ordering.
🛠️ Author fix: Keep INV-12 read-gate-first ordering; fix only the dual-bucket debit logic and tests.
🚫 Out of scope: Re-mounting issue routes with router-level write_braked.
🔵 P3 — New in-handler write_rate_limiter debits omit Retry-After while middleware 429s advertise window delay
📍 Where: crates/gitlawb-node/src/api/issues.rs (create_issue_comment ~185–190; close_issue write-bucket arms ~298–306)
💥 What fails: These paths return AppError::TooManyRequests with no retry-after header. This PR’s router brakes use check_retry and attach window-derived Retry-After on 429.
🔎 Root cause: Handler paths call check() (bool) instead of check_retry, and TooManyRequests maps to status-only.
📜 Stated contract:
This PR extended limiter and middleware 429 behavior to return and test window-derived
Retry-After(rate_limit.rs, receive-pack advert and middleware tests).
🏷️ Attribution: PR-introduced incomplete — only the write_rate_limiter debits added or wired on this branch for issue comment/close; the pre-existing close_issue_rate_limiter-only path at merge-base already omitted Retry-After and is not part of this finding.
📌 In this PR:
create_issue_commentpost-read-gate write debitclose_issueshared write-bucket debit (not the close-only cap alone)
🔒 Unchanged on main: Historical handler 429 sites that never debited write_rate_limiter; global redesign of every TooManyRequests variant.
🔧 Required correction: On those write-bucket rejections, use check_retry (or return the same too_many_requests(Duration) helper middleware uses) so 429 carries retry-after; one regression on comment or close write-cap is enough.
🛠️ Author fix: Local helper is fine; do not retrofit unrelated handlers in this PR.
🚫 Out of scope: Adding Retry-After to merge-base close-cap-only 429s or other untouched limiters.
…ry-After on write-bucket 429s (#196) close_issue debited the write bucket and then the close bucket, so a concurrent close taking the last close slot between the probe and the check left the write slot spent with no close done. Drop the probes: debit write, then close, and refund the write slot when close refuses. The in-handler write-bucket refusals in close_issue and create_issue_comment now return AppError::RateLimited, which maps to 429 with a window-derived Retry-After like the router middleware's 429s. The close-cap-only refusal keeps its plain 429.
Fixed in 11cadb9. I dropped the probes and
Fixed in 11cadb9. The write-bucket refusals in |
jatmn
left a comment
There was a problem hiding this comment.
🟢 Thanks for the contribution. I do not see any actionable issues from my review.
Merge readiness
Head 11cadb938260dc8d6ddd2310e3930014a4e5ddb0 is mergeable but two commits behind live origin/main (5a7576b682e1fc2566c25b09a1b69edf85e836cb). Rebase onto current main before merge so any recent main fixes ride with this branch.
Required test (stable) and test (beta) failed on run 43337356159:
reconciliation::tests::sweep_persists_last_attempted_not_planned_tail_on_mid_batch_narrow(stable)ipfs_pin::tests::discovery_record_elapsed_leaves_neither_row(beta)
Neither crates/gitlawb-node/src/reconciliation.rs nor crates/gitlawb-node/src/ipfs_pin.rs appears in this PR’s three-dot diff. Treat these as required-check failures that must be green on the PR head (re-run after rebase, or chase flake/regression on main if they reproduce there). Other required checks on that run (fmt+clippy, deny harness job aside from the single beta failure, MSRV, docker smoke, CodeQL matrix) were green.
Prior jatmn review items from 2026-11-11 (close/merge compensating upload rollback, purge archive-after-disk-refusal, execute-time symlink recheck, write-limiter sweep regression) are addressed on current head per code inspection and the author’s follow-up commits through 11cadb93 (including write-bucket Retry-After on in-handler refusals and close-cap refund for the single-request dual-bucket window).
Summary
Adds a per-IP rate brake on the authenticated non-creation write surface and a
purge-spamadmin subcommand for cleaning up an empty spam burst.Motivation & context
Creation-surface abuse resistance. Before this, the write route groups had no shared per-IP limit, so a single-source flood on them was uncounted.
No linked issue.
Kind of change
What changed
Crate touched: gitlawb-node.
Two related pieces of the creation-surface abuse-resistance work:
A per-IP write-surface rate brake. Every authenticated non-creation write sink now sits behind a per-client-IP flood brake on its own bucket: pull/issue comments, issue close, labels, stars, merges, protect/unprotect, replicas, webhook create/delete, visibility (including the GET on that group), agent deregister, tasks, bounties, profile, and every HTTP
/graphqlrequest. Issue comment and issue close debit the bucket inside the handler, after the read gate, rather than in router middleware; issue close also draws on its ownGITLAWB_CLOSE_ISSUE_RATE_LIMITbucket, and a close refused by either cap spends nothing from the other. Previously these route groups (the webhook routes among them) mounted with no shared write limit (issue close had only its own bucket), so a single-source flood on any of them was uncounted by the write budget (only repo/agent creation, git-push, and peer-sync were braked). The bucket is configured byGITLAWB_WRITE_RATE_LIMIT(default 600/hr,0disables), keyed on the resolved client IP via the existingGITLAWB_TRUSTED_PROXYboundary.A
purge-spamadmin subcommand for cleaning up a specific empty spam-burst that predates the mirror bot. Dry-run by default;--executedeletes.Key decisions
sync_triggervspeer_writesplit). A shared bucket would also self-throttle a legitimate session doing repo + issues + comments.did:keyfarm never trips a per-DID limit (the same reasongit-receive-packdeliberately omits it), and a per-DID brake would false-positive busy legitimate agents./graphqlrequest;/graphql/wssubscriptions stay unbraked.MutationRootwas the largest previously-uncounted write vector.How a reviewer can verify
cargo test -p gitlawb-node --lib cargo fmt --all --check cargo clippy --workspace --all-targets -- -D warningscargo test --workspacegreen;cargo fmt --all --checkandcargo clippy --workspace --all-targets -D warningsclean.TrustedProxycoverage: a spoofed leftmostX-Forwarded-Forhop cannot rotate the bucket key, distinct trusted hops get distinct buckets, and an absent header falls back to the socket peer.Before you request review
cargo test --workspacepasses locallycargo fmt --allandcargo clippy --workspace --all-targets -- -D warningsare cleanfeat(...),fix(...),docs(...)).env.exampleupdated if behavior or config changed (or N/A)Protocol & signing impact
N/A: no change to DID/
did:key, signatures, UCAN, ref certs, or P2P wire formats.Notes for reviewers
This is the write-surface brake coverage plus the burst cleanup. iCaptcha enforcement, storage/size quotas, and moving the gate toward enforce-by-default are separate follow-ups, not in this PR.
Summary by CodeRabbit
purge-spamadmin command with a dry-run mode and optional execution.GITLAWB_WRITE_RATE_LIMIT=0to disable it; GraphQL WebSocket traffic is unaffected.Retry-Aftervalue..env.example.